[8.x] Add upsert to Eloquent and Base Query Builders #34698

paras-malhotra · 2020-10-06T09:33:01Z

Motivation
Laravel already supports bulk inserts, bulk insert/ignores. It's only logical to also support bulk upserts, which are already supported on frameworks such as Rails.

Bulk Inserts Already Supported

Bulk inserts: errors out on any row with invalid data

DB::table('users')->insert([
    ['email' => 'taylor@example.com', 'votes' => 0],
    ['email' => 'dayle@example.com', 'votes' => 0],
]);

Bulk insert or ignore: rows with invalid data are ignored

DB::table('users')->insertOrIgnore([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
]);

Bulk Upsert (This PR)

This PR supports bulk upserts (insert and on duplicate key, update) like so:

DB::table('users')->upsert([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
], 'email');

The first argument is the values to insert/update and the second argument is the column(s) that uniquely identify records. All databases except SQL Server require these columns to have a PRIMARY or UNIQUE index. This is similar to Rails upsert_all without the returning support.

Eloquent upsets are also supported like so:

User::upsert([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
], 'email');

If the model uses the updated_at timestamp, upsert() will also add the updated_at timestamp columns. Inspired from https://github.com/staudenmeir/laravel-upsert with some slight modifications. Ping @staudenmeir

rodrigopedra · 2020-10-06T14:51:36Z

Usage of VALUES() on MySQL is deprecated starting on MySQL 8.0.20.

Reference:

https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html

The use of VALUES() to refer to the new row and columns is deprecated beginning with MySQL 8.0.20, and is subject to removal in a future version of MySQL. Instead, use row and column aliases, as described in the next few paragraphs of this section.

The sample below is from the link above showcasing the new syntax to use:

INSERT INTO t1 (a,b,c) VALUES (1,2,3),(4,5,6) AS new
  ON DUPLICATE KEY UPDATE c = new.a+new.b;

taylorotwell · 2020-10-06T14:54:19Z

@rodrigopedra does that new syntax work in MySQL 5.7? Though I guess it doesn't really matter as this feature could require MySQL 8.0.

rodrigopedra · 2020-10-06T15:05:45Z

Old syntax works on both MySQL versions (5 and 8).

New syntax was introduced in MySQL 8.0.19, from the link on my previous message:

Beginning with MySQL 8.0.19, it is possible to use an alias for the row...

To support both versions I think we can keep the old syntax around as the docs says: "...and is subject to removal in a future version of MySQL". No removal timeline was set yet.

In general it takes a while to MySQL really remove deprecated features (for example the GROUP BY ... DESC was deprecated in 5.7 (https://dev.mysql.com/doc/refman/5.7/en/select.html) but was removed only in 8.0 and noted in the upgrade guide.

So we could use the old syntax for now and handle this case when it actually gets removed.

EDIT: corrected the deprecation example about GROUP BY...DESC as it was actually deprecated in MySQL 5.7

paras-malhotra · 2020-10-06T15:20:59Z

Yeah, it's still supported but deprecated beginning 8.0.20. It isn't removed yet and will be around for a while. So to support both mySQL 5 and 8, it's best to continue using values.

Besides, we're already using VALUES() in all insert operations including insert and insertOrIgnore.

rodrigopedra · 2020-10-06T15:29:45Z

Actually @paras-malhotra, the usage of VALUES as it is used now in insert and insertOrIgnore is not deprecated and is part of the SQL standard, it is a clause of the INSERT SQL statement.

What is deprecated is the usage of the VALUES() special function on INSERT...ON DUPLICATE KEY UPDATE statement.

https://dev.mysql.com/doc/refman/8.0/en/miscellaneous-functions.html#function_values

paras-malhotra · 2020-10-06T15:35:31Z

Ohh wow, I didn't know that 😮. Okay, I'll probably look out for the removal notice since row/column aliases don't work in 5.7. @taylorotwell if you'd like I can implement the new way (row/column aliases), but I'd recommend we go with values for now as it's still supported in both versions.

rodrigopedra · 2020-10-06T15:40:24Z

No problem, maybe one of the reasons to deprecate that function is the confusion with the SQL clause usage.

MariaDB took a similar step, but they renamed the VALUES() function to VALUE() (https://mariadb.com/kb/en/values-value/).

I think this feature is a great addition to the framework and that for new it is best to keep the VALUES() usage until it actually gets removed from MySQL. Maybe by then we can remove MySQL 5.x support.

mpskovvang · 2020-10-13T18:15:03Z

Great addition! This is the first I add to new projects myself.

However, this PR misses the option to do partial updates. You don't want to keep overwriting e.g. created_at.

@paras-malhotra do you know if partial updates is supported by other than MySQL and PostgreSQL? Thanks for contributing this to the core.

Perhaps a third argument with either an indexed array of column names or an associate array with key-value. The default is to update all columns.

paras-malhotra · 2020-10-13T18:40:35Z

@mpskovvang absolutely, that was the follow up PR: #34712 😄

mpskovvang · 2020-10-13T20:51:22Z

@mpskovvang absolutely, that was the follow up PR: #34712 😄

Did't see that. Spot on! Thanks! Much appreciated. 🙏

webbby · 2020-10-15T19:40:49Z

Does the feature support columns combination that uniquely identifies records?

For example

App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);

departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

@paras-malhotra

Pulls in the work by @paras-malhotra in laravel/framework#34698 & laravel/framework#34712 for use in October CMS.

Added support for upsert in Laravel 8.10.0 laravel/framework#34698

Based off implementation done in laravel/framework#34698 and laravel/framework#34712.

@paras-malhotra

Pulls in the work by @paras-malhotra in https:ithub.com/laravel/framework/pull/34698 & laravel/framework#34712 for use in October CMS. Co-authored-by: Ben Thomson <git@alfreido.com>

shankhadevpadam · 2020-11-04T07:42:12Z

I have post meta table with fields [id, post_id, meta_key, meta_value]. i want to bulk update a meta values. But It repeatedly create a new row when using upsert method while updating form every time.

foreach ($metas as $key => $value) {
    $arr[] = [
        'post_id' => $post->id,
        'meta_key' => $key,
        'meta_value' => $value ?? '',
    ];
}

Postmeta::upsert($arr, ['post_id', 'meta_key'], ['meta_value']);

but while using updateOrCreate method it work fine.

foreach ($metas as $key => $value) {
    Postmeta::updateOrCreate(['post_id' => $post->id, 'meta_key' => $key], ['meta_value' => $value ?? '']);
}

paras-malhotra · 2020-11-04T08:10:58Z

@shankhadevpadam all databases except SQL Server require the columns in the second argument of the upsert method to have a "primary" or "unique" index. If that doesn't solve your problem, please create a new issue with steps to replicate.

eusonlito · 2020-11-30T09:19:35Z

Only as reference, is important understand than INSERT INTO ... ON CONFLICT will update the internal reference to increment primary key on related table because the increment has to happen before the insert is attempted. Then, if you use the upsert method as a regular insert or update action, your ID will be something like:

-[ RECORD 284 ]
id | 19039
-[ RECORD 285 ]
id | 19221
-[ RECORD 286 ]
id | 19306
-[ RECORD 287 ]
id | 19397
-[ RECORD 288 ]
id | 19784
-[ RECORD 289 ]
id | 19828
-[ RECORD 290 ]
id | 20161
-[ RECORD 291 ]
id | 20768
-[ RECORD 292 ]
id | 20799
-[ RECORD 293 ]
id | 21287
-[ RECORD 294 ]
id | 21770
-[ RECORD 295 ]
id | 22205
-[ RECORD 296 ]
id | 22401
-[ RECORD 297 ]
id | 23099
-[ RECORD 298 ]
id | 23727
-[ RECORD 299 ]
id | 23820
-[ RECORD 300 ]
id | 24190
-[ RECORD 301 ]
id | 24653
-[ RECORD 302 ]
id | 24744
-[ RECORD 303 ]
id | 25054
-[ RECORD 304 ]
id | 25295

fractalf · 2021-01-11T13:45:33Z

I noticed that upsert will add values for created_at and updated_at, but insert does not do this.

Is that intended?

My preference would be that insert also has this behaviour

ejntaylor · 2021-03-25T15:56:29Z

Does the feature support columns combination that uniquely identifies records?

For example
App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);
departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

Doesn't look like this is the case - anyone had luck with this?

rognales · 2021-09-07T10:03:35Z

When issuing a mass update or delete query via Eloquent, the saved, updated, deleting, and deleted model events will not be dispatched for the affected models. This is because the models are never actually retrieved when performing mass updates or deletes

This statements applies to UPSERT as well?

Because the created_at and updated_at did get filled by Eloquent

mdobydullah · 2021-12-03T07:41:00Z

Does the feature support columns combination that uniquely identifies records?

For example
App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);
departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

This feature is badly needed. Do you have any solutions regarding this issue?

paras-malhotra added 3 commits October 6, 2020 14:37

[8.x] Add upsert to Eloquent and Base Query Builders

e33b538

fix styleci and base grammer

8eb6522

fix styleci

4139c10

taylorotwell merged commit 1553d1c into laravel:8.x Oct 6, 2020

paras-malhotra deleted the bulk_upsert branch October 6, 2020 15:46

This was referenced Oct 6, 2020

Possibly merge to the core staudenmeir/laravel-upsert#23

Closed

[8.x] Document the new upsert method laravel/docs#6466

Merged

[8.x] Support created_at timestamps on upserts #34712

Merged

ejunker mentioned this pull request Oct 13, 2020

Use INSERT...ON DUPLICATE KEY UPDATE optimization julienbourdeau/route-usage#19

Open

LukeTowers added a commit to octobercms/library that referenced this pull request Oct 19, 2020

Add support for QueryBuilder upsert() method.

a49a954

Pulls in the work by @paras-malhotra in laravel/framework#34698 & laravel/framework#34712 for use in October CMS.

LukeTowers mentioned this pull request Oct 19, 2020

Add support for QueryBuilder upsert() method. octobercms/library#531

Merged

sunaoka added a commit to sunaoka/laravel-postgres-extension that referenced this pull request Oct 20, 2020

Support for Laravel 8.10.0

8386557

Added support for upsert in Laravel 8.10.0 laravel/framework#34698

bennothommo added a commit to octobercms/library that referenced this pull request Oct 20, 2020

Add unit tests for upsert() query builder method.

4e89741

Based off implementation done in laravel/framework#34698 and laravel/framework#34712.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.x] Add upsert to Eloquent and Base Query Builders #34698

[8.x] Add upsert to Eloquent and Base Query Builders #34698

paras-malhotra commented Oct 6, 2020

rodrigopedra commented Oct 6, 2020

taylorotwell commented Oct 6, 2020 •

edited

Loading

rodrigopedra commented Oct 6, 2020 •

edited

Loading

paras-malhotra commented Oct 6, 2020 •

edited

Loading

rodrigopedra commented Oct 6, 2020 •

edited

Loading

paras-malhotra commented Oct 6, 2020

rodrigopedra commented Oct 6, 2020

mpskovvang commented Oct 13, 2020

paras-malhotra commented Oct 13, 2020

mpskovvang commented Oct 13, 2020

webbby commented Oct 15, 2020 •

edited

Loading

shankhadevpadam commented Nov 4, 2020

paras-malhotra commented Nov 4, 2020

eusonlito commented Nov 30, 2020

fractalf commented Jan 11, 2021 •

edited

Loading

ejntaylor commented Mar 25, 2021

rognales commented Sep 7, 2021 •

edited

Loading

mdobydullah commented Dec 3, 2021 •

edited

Loading

[8.x] Add upsert to Eloquent and Base Query Builders #34698

[8.x] Add upsert to Eloquent and Base Query Builders #34698

Conversation

paras-malhotra commented Oct 6, 2020

rodrigopedra commented Oct 6, 2020

taylorotwell commented Oct 6, 2020 • edited Loading

rodrigopedra commented Oct 6, 2020 • edited Loading

paras-malhotra commented Oct 6, 2020 • edited Loading

rodrigopedra commented Oct 6, 2020 • edited Loading

paras-malhotra commented Oct 6, 2020

rodrigopedra commented Oct 6, 2020

mpskovvang commented Oct 13, 2020

paras-malhotra commented Oct 13, 2020

mpskovvang commented Oct 13, 2020

webbby commented Oct 15, 2020 • edited Loading

shankhadevpadam commented Nov 4, 2020

paras-malhotra commented Nov 4, 2020

eusonlito commented Nov 30, 2020

fractalf commented Jan 11, 2021 • edited Loading

ejntaylor commented Mar 25, 2021

rognales commented Sep 7, 2021 • edited Loading

mdobydullah commented Dec 3, 2021 • edited Loading

taylorotwell commented Oct 6, 2020 •

edited

Loading

rodrigopedra commented Oct 6, 2020 •

edited

Loading

paras-malhotra commented Oct 6, 2020 •

edited

Loading

rodrigopedra commented Oct 6, 2020 •

edited

Loading

webbby commented Oct 15, 2020 •

edited

Loading

fractalf commented Jan 11, 2021 •

edited

Loading

rognales commented Sep 7, 2021 •

edited

Loading

mdobydullah commented Dec 3, 2021 •

edited

Loading